Skip to content

CLDSRV-917: allow writing objects directly to a cold location - #6264

Open
francoisferrand wants to merge 1 commit into
development/9.4from
improvement/CLDSRV-917-direct-to-cold
Open

CLDSRV-917: allow writing objects directly to a cold location#6264
francoisferrand wants to merge 1 commit into
development/9.4from
improvement/CLDSRV-917-direct-to-cold

Conversation

@francoisferrand

@francoisferrand francoisferrand commented Aug 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Users had no way to store an object in a cold location without first writing it hot and waiting for a lifecycle transition rule to kick in — impractical when the intent is already known at write time.

PutObject, CreateMultipartUpload and CopyObject now accept the name of a cold location in x-amz-storage-class. The data is still written to the hot location as usual, but the object is stamped with the cold storage class and flagged as transition-in-progress, so the lifecycle queue populator picks it up from the oplog and drives the transition from there. Cloudserver writes no Kafka message itself, and a requeue keeps the flag set.

Resulting metadata for such a write:

field value
x-amz-storage-class the cold location
dataStoreName the hot location (unchanged)
archive absent
x-amz-scal-transition-in-progress true
x-amz-scal-transition-time now
originOp the real S3 event (Put / CompleteMultipartUpload / Copy)

Gated by a new off-by-default enableDirectToCold option (ENABLE_DIRECT_TO_COLD). Which identities may use a given storage class is left to the existing s3:x-amz-storage-class IAM condition key, so there is no new authorization code here.

Also drops the dead CLDSRV-639 lowercase/uppercase storage class juggling in CreateMultipartUpload — the value is validated beforehand, so it can just be stored as-is. That code could never match anyway, since validStorageClasses only holds uppercase values.

Issue: CLDSRV-917

@bert-e

bert-e commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-917 contains:

  • 9.5.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.3

Please check the Fix Version/s of CLDSRV-917, or the target
branch of this pull request.

@codecov

codecov Bot commented Aug 24, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 13 Tests Failed:

Tests completed Failed Passed Skipped
9914 13 9901 0
View the top 3 failed test(s) by shortest run time 
should handle abort during concurrent CompleteMPU without corruption::Abort MPU - Race Conditions With v4 signature should handle abort during concurrent CompleteMPU without corruption
Stack Traces | 0.123s run time 
No MPU metadata should remain

1 !== 0

"before each" hook: beforeEachF for "should complete MPU by concatenating the parts in the following order: 8,1000,3"::More MPU tests With v4 signature "before each" hook: beforeEachF for "should complete MPU by concatenating the parts in the following order: 8,1000,3"
Stack Traces | 30.1s run time 
Connection timed out after 30000 ms
"before all" hook for "should return the total size of the object when --part-number is not used"::Part size tests with object head With default signature "before all" hook for "should return the total size of the object when --part-number is not used"
Stack Traces | 39.4s run time 
Connection timed out after 30000 ms
View the full list of 12 ❄️ flaky test(s) 
"after each" hook for "should batch delete 1000 objects"::Multi-Object Delete Success "after each" hook for "should batch delete 1000 objects"

Flake rate in main: 100.00% (Passed 0 times, Failed 7 times)

Stack Traces | 2.52s run time 
The bucket you tried to delete is not empty.
"after each" hook for "should handle abort during concurrent CompleteMPU without corruption"::Abort MPU - Race Conditions With v4 signature "after each" hook for "should handle abort during concurrent CompleteMPU without corruption"

Flake rate in main: 100.00% (Passed 0 times, Failed 8 times)

Stack Traces | 0.03s run time 
The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.
"before each" hook for "should batch delete 1000 objects"::Multi-Object Delete Success "before each" hook for "should batch delete 1000 objects"

Flake rate in main: 100.00% (Passed 0 times, Failed 7 times)

Stack Traces | 5.26s run time 
Socket timed out without establishing a connection within 5000 ms
"before each" hook for "should grant read access"::With default signature PUT Bucket ACL using predefined groups - authenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.123s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With default signature PUT Bucket ACL using predefined groups - unauthenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.137s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With default signature PUT Object ACL using predefined groups - authenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.051s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With default signature PUT Object ACL using predefined groups - unauthenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.225s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With v4 signature PUT Bucket ACL using predefined groups - authenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.085s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With v4 signature PUT Bucket ACL using predefined groups - unauthenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.134s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With v4 signature PUT Object ACL using predefined groups - authenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.044s run time 
We encountered an internal error. Please try again.
"before each" hook for "should grant read access"::With v4 signature PUT Object ACL using predefined groups - unauthenticated request "before each" hook for "should grant read access"

Flake rate in main: 100.00% (Passed 0 times, Failed 37 times)

Stack Traces | 0.113s run time 
We encountered an internal error. Please try again.
should allow action on an object specified in the policy::With default signature Bucket policies with resource statement should allow action on an object specified in the policy

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 40s run time 
Timeout of 40000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (.../test/bucket/bucketPolicyWithResourceStatements.js)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@francoisferrand francoisferrand self-assigned this Aug 24, 2026
@francoisferrand
francoisferrand force-pushed the improvement/CLDSRV-917-direct-to-cold branch from 555417e to 73b5a05 Compare August 28, 2026 11:08
@francoisferrand
francoisferrand requested review from a team, SylvainSenechal, delthas and maeldonn and removed request for SylvainSenechal August 28, 2026 11:19
@francoisferrand
Allow writing objects directly to a cold location
fa0442f 
Users had no way to store an object in a cold location without first
writing it hot and waiting for a lifecycle transition rule to kick in,
which is impractical when the intent is known upfront.

PutObject, CreateMultipartUpload and CopyObject now accept the name of a
cold location in x-amz-storage-class. The data is still written to the
hot location as usual, but the object is stamped with the cold storage
class and flagged as transition-in-progress, so the lifecycle queue
populator picks it up from the oplog and drives the transition. No Kafka
message is written by cloudserver itself, and a requeue keeps the flag.

This is gated by a new off-by-default `enableDirectToCold` option
(ENABLE_DIRECT_TO_COLD). Which identities may use a given storage class
is left to the existing s3:x-amz-storage-class IAM condition key.

Also drops the dead CLDSRV-639 lowercase/uppercase storage class
handling in CreateMultipartUpload: the value is validated beforehand, so
it can be stored as-is.

Issue: CLDSRV-917
@francoisferrand
francoisferrand force-pushed the improvement/CLDSRV-917-direct-to-cold branch from 73b5a05 to fa0442f Compare August 28, 2026 11:25
@francoisferrand francoisferrand removed their assignment Aug 28, 2026
@bert-e

bert-e commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-917 contains:

  • 9.5.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.3

  • 9.5.0

Please check the Fix Version/s of CLDSRV-917, or the target
branch of this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@delthas delthas delthas approved these changes
@maeldonn maeldonn maeldonn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@francoisferrand @bert-e @delthas @maeldonn